Image decoding method, image coding method, image decoding apparatus, image coding apparatus, program, and integrated circuit

ABSTRACT

An image decoding method which can improve both image quality and coding efficiency is an image decoding method for decoding a coded stream which includes a plurality of processing units and a header for the processing units, the coded stream being generated by coding a moving picture, the processing units including at least one processing unit layered to be split into a plurality of smaller processing units, the image decoding method including specifying a hierarchical layer having a processing unit in which a parameter necessary for decoding is stored, by parsing hierarchy depth information stored in the header, and decoding the processing unit using the parameter stored in the processing unit located at the specified hierarchical layer.

TECHNICAL FIELD

The present invention relates to an image coding method for codingimages or moving pictures included in multimedia data and an imagedecoding method for decoding coded images or moving pictures.

BACKGROUND ART

In video compressions standards like MPEG-1, MPEG-2, MPEG-4, or MPEG-4AVC, a compressed picture is usually divided into rectangle units called“macroblocks”. A macroblock is usually defined as a two-dimensionalblock of image samples. The image samples have a width of 16 pixels anda height of 16 pixels for luminance samples. The compression ratio forthe macroblock is controlled by a quantization scale parameter for eachmacroblock. The quantization scale parameter determines the level ofquantization to be applied to all the frequency coefficients. Thequantization scale parameter is usually coded as a difference value fromthe quantization scale parameter of the previous macroblock in cordingorder, and is stored in a compressed macroblock header.

In new video standards under development, for example, the HighEfficiency Video Coding (HEVC) standard by the MPEG standardizationbodies, it is suggested that dividing the picture into large units canimprove the coding efficiency of the compressed video (for example,refer to Non Patent Literature 1). In other words, a picture can bedivided into coding units (CU) where each coding unit has a size thatcan be much larger than a macroblock. For example, the coding unit sizecan be 128 pixels by 128 pixels for luminance samples, which isapproximately 64 times larger than a macroblock.

A large coding unit can be sub-divided into smaller units (sub codingunits) to achieve better coding efficiency. Each coding unit or subcoding unit has three main components. The main components are a codingunit header, a prediction unit (PU), and a transform unit (TU).

FIG. 1 is a diagram showing the structure of compressed picture havingcoding units.

As shown in FIG. 1, a picture D100 includes a header (hereinafterreferred to as picture header) and a body. The picture header includesparameters related to the picture (picture parameters) while the bodyincludes compressed samples of a picture. Moreover, the body includescoding units such as coding units D102 and D104, and some of the codingunits are divided into sub coding units. For example, the coding unitD102 is divided into sub coding units D106, and one of the sub codingunits 106 is further divided into smaller sub coding units D108. Thecoding unit D104 or sub coding unit D108 has three main components. Morespecifically, the coding unit D104 includes a coding unit header D116, aprediction unit D118, and a transform unit D120 as the three maincomponents. The sub coding unit D108 has a sub coding unit header D110,a prediction unit D112, and a transform unit D114 as the three maincomponents. As shown in FIG. 1, a transform unit D120 is divided intosmall sub transform units D122, and one of the sub transform units D122is divided into smaller sub transform units D124. The smallest transformunits (sub transform units) D114 and D124 includes the quantizedcoefficients of a block, which requires a quantization scale parameterfor the inverse quantization process of the coefficients.

CITATION LIST Non Patent Literature

[NPL 1]

“Test Model under Consideration” Joint Collaborative Team on VideoCoding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 2ndMeeting: Geneva, CH, 21-28 July, 2010, Document: JCTVC-B205

SUMMARY OF INVENTION Technical Problem

However, in the image decoding method and the image coding methodaccording to the above described Non Patent Literature 1, there is aproblem that image quality and coding efficiency cannot be improvedsimultaneously. In other words, a parameter such as a quantization scaleparameter necessary for coding and a coding process is stored in apicture header such that the parameter is applied to the whole body ofthe picture D100. As a result, it is impossible for image quality to beadjusted for each of the small processing units such as the sub codingunit, the prediction unit, the sub prediction unit, the transform unit,or the sub transform unit. Moreover, the amount of coding is large whena parameter is stored for each of the processing units such that theparameter is applied to each of the smallest processing units.

Therefore, the present invention is conceived in view of the problem,and an object of the present invention is to provide an image decodingmethod and an image coding method for simultaneously improving image andcoding efficiency.

Solution to Problem

In order to attain the above described goal, an image decoding methodaccording to an aspect of the present invention is an image decodingmethod for decoding a coded stream which includes a plurality ofprocessing units and a header for the processing units, the coded streambeing generated by coding a moving picture, the processing unitsincluding at least one processing unit layered to be split into aplurality of smaller processing units, the image decoding methodincluding: specifying a hierarchical layer having a processing unit inwhich a parameter necessary for decoding is stored, by parsing hierarchydepth information stored in the header; and decoding the processing unitusing the parameter stored in the processing unit located at thespecified hierarchical layer.

With this, the processing unit is hierarchically layered. When aparameter is stored in each small processing unit located at a lowerhierarchical layer, the small processing units can be decoded byapplying a different parameter to each of the small processing units. Asa result, image quality can be improved. Moreover, since thehierarchical layer having a processing unit in which a parameternecessary for decoding are stored is specified by parsing hierarchydepth information, it is possible for the hierarchical layer to be setat an arbitrary hierarchical layer instead of being limited to thelowest hierarchical layer. Therefore, the amount of coding for allparameters included in a coded stream can be reduced compared with thecase where a parameter is stored for each of the smallest processingunits located at the lowest hierarchical layer, and coding efficiencycan be improved. With this, image quality and coding efficiency can beimproved simultaneously. Moreover, since by parsing the hierarchy depthinformation, a hierarchical layer having a processing unit in which aparameter is stored is specified, it is possible to reduce the burden ofa process of searching the processing unit in which the parameter isstored.

Moreover, the coded stream is generated by coding which includesorthogonal transform and quantization, the processing unit is layered tobe smaller in a direction from a higher level to a lower level, a codingunit exists as a largest processing unit at a highest hierarchicallayer, and a transform unit exists as a processing unit smaller than thecoding unit at a lower hierarchical layer that is deeper than thehighest hierarchical layer, the parameter is a quantization parameterapplied to the transform unit, the hierarchy depth information indicatesa lower hierarchical layer that is deeper than the highest hierarchicallayer, and (i) a hierarchical layer indicated by the hierarchy depthinformation or (ii) a hierarchical layer which is higher than thehierarchical layer and is other than the highest hierarchical layer isspecified, when specifying a hierarchical layer having a processing unitin which the quantization parameter is stored.

With this, it is possible for image quality in quantization by aquantization parameter and coding efficiency in the quantizationparameter to be improved simultaneously.

Moreover, the header may be a picture header for a picture including theprocessing units, and the hierarchy depth information may be stored inthe picture header.

With this, a hierarchy having a processing unit in which a parameternecessary for decoding is stored can be identified as a commonhierarchical layer for the whole picture.

Moreover, when the processing unit is decoded, the quantizationparameter located, within the processing unit, after a transformcoefficient generated by the orthogonal transform and quantization maybe used.

With this, since the quantization parameter is stored only when thereare transform coefficients, the quantization parameter is not storedwhen there are no transform coefficients and coding efficiency can beimproved.

Moreover, in order to achieve the above mentioned goal, an image codingmethod according to an aspect of the present invention is an imagecoding method for generating, by coding a moving picture, a coded streamwhich includes a plurality of processing units and a header for theprocessing units, the processing units including at least one processingunit layered to be split into a plurality of smaller processing units,the image coding method including: coding the moving picture; writing,into the header, hierarchy depth information for specifying ahierarchical layer having a processing unit in which a parameternecessary for decoding is stored; and writing the parameter into theprocessing unit located at the hierarchical layer specified by thehierarchy depth information.

With this, when the processing unit is hierarchically layered, aparameter can be written which is different for each of the smallprocessing units that are located at the low hierarchical layers. As aresult, the image decoding apparatus can decode the processing units byapplying a different parameter to each of the small processing units,and therefore image quality can be improved. Moreover, by writing, intoa header, the hierarchy depth information for specifying thehierarchical layer having a processing unit in which a parameternecessary for decoding is stored, the hierarchical layer can be notifiedto the image decoding apparatus. Therefore, an arbitrary hierarchicallayer can be set without limiting the hierarchical layer to the lowesthierarchical layer. Therefore, the amount of coding for all parametersincluded in a coded stream can be reduced compared with the case where aparameter is stored for each of the smallest processing unit located atthe lowest hierarchical layer, and coding efficiency can be improved.With this, image quality and coding efficiency can be improvedsimultaneously.

Moreover, orthogonal transform and quantization are performed on themoving picture when the moving picture is coded, the processing unit islayered to be smaller in a direction from a higher level to a lowerlevel, a coding unit exists as a largest processing unit at a highesthierarchical layer, and a transform unit exists as a processing unitsmaller than the coding unit at a lower hierarchical layer that isdeeper than the highest hierarchical layer, the parameter is aquantization parameter applied to the transform unit, the hierarchydepth information indicates a hierarchical layer that is lower than thehighest hierarchical layer, and the parameter is written into aprocessing unit at (i) a hierarchical layer indicated by the hierarchydepth information or (ii) a hierarchical layer which is higher than thehierarchical layer and is other than the highest hierarchical layer,when the quantization parameter is written.

With this, it is possible for image quality in quantization by aquantization parameter and coding efficiency in the quantizationparameter to be improved simultaneously.

Moreover, the header is a picture header for a picture including theprocessing units, and the hierarchy depth information may be writteninto the picture header when the hierarchy depth information is written.

With this, a hierarchical layer having a processing unit in which aparameter necessary for decoding is stored can be commonly set for thewhole picture.

Moreover, when the quantization parameter is written, the quantizationparameter may be written within the processing unit, after a transformcoefficient generated by the orthogonal transform and quantization.

With this, it is possible for a quantization parameter to be writtenonly when there are transform coefficients, and coding efficiency can beimproved.

It should be noted that the present invention can be implemented as theabove described image decoding method and image coding method. It canalso be implemented as an apparatus for coding or decoding an image, anintegrated circuit, a program for decoding or coding an image accordingto the methods, and a recording medium having the program storedthereon.

Advantageous Effects of Invention

The image decoding method and the image coding method according to thepresent invention can improve image quality and coding efficiencysimultaneously.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing the configuration of a conventional codedstream.

FIG. 2 is a block diagram showing the configuration of an image codingapparatus according to Embodiment 1 of the present invention.

FIG. 3 is a block diagram showing the configuration of an image decodingapparatus according to Embodiment 1 of the present invention.

FIG. 4 is an illustration diagram for describing a multi-hierarchicalblock structure.

FIG. 5 is a diagram showing the configuration of a coded stream to begenerated by TMuC software.

FIG. 6A is a diagram showing the configuration of a coded streamaccording to Embodiment 1 of the present invention.

FIG. 6B is a diagram showing the configuration of a coded streamaccording to Embodiment 1 of the present invention.

FIG. 6C is a diagram showing the configuration of a coded streamaccording to Embodiment 1 of the present invention.

FIG. 7 is a diagram showing the configuration of another coded streamaccording to Embodiment 1 of the present invention.

FIG. 8A is a diagram showing the configuration of a still another codedstream according to Embodiment 1 of the present invention.

FIG. 8B is a diagram showing the configuration of a still another codedstream according to Embodiment 1 of the present invention.

FIG. 9A is a diagram showing the storage position ofMax_quantization_unit_hierarchy_depth according to Embodiment 1 of thepresent invention.

FIG. 9B is a diagram showing the storage position ofMax_quantization_unit_hierarchy_depth according to Embodiment 1 of thepresent invention.

FIG. 10A is a diagram showing a delta quantization scale parameteraccording to Embodiment 1 of the present invention.

FIG. 10B is a diagram showing a quantization dead zone offset parameteraccording to Embodiment 1 of the present invention.

FIG. 10C is a diagram showing an index according to Embodiment 1 of thepresent invention.

FIG. 10D is a diagram showing a quantization offset parameter accordingto Embodiment 1 of the present invention.

FIG. 11 is a flowchart showing decoding of delta QP by the imagedecoding apparatus according to Embodiment 1 of the present invention.

FIG. 12 is a flowchart showing computation of QP by the image decodingapparatus according to Embodiment 1 of the present invention.

FIG. 13 is a flowchart showing decoding by the image decoding apparatusaccording to Modification 1 of Embodiment 1 of the present invention.

FIG. 14 is a flowchart showing coding by the image coding apparatusaccording to Modification 1 of Embodiment 1 of the present invention.

FIG. 15A is a flowchart showing decoding by the image decoding apparatusaccording to Modification 2 of Embodiment 1 of the present invention.

FIG. 15B is a flowchart showing decoding by the image decoding apparatusaccording to Modification 2 of Embodiment 1 of the present invention.

FIG. 16A is a flowchart showing coding by the image coding apparatusaccording to Modification 2 of Embodiment 1 of the present invention.

FIG. 16B is a flowchart showing coding by the image coding apparatusaccording to Modification 2 of Embodiment 1 of the present invention.

FIG. 17A is a flowchart showing an image decoding method according tothe present invention.

FIG. 17B is a flowchart showing an image coding method according to thepresent invention.

FIG. 18A is a diagram showing a syntax of sequence header according toEmbodiment 1 of the present invention.

FIG. 18B is a diagram showing a syntax of picture header according toEmbodiment 1 of the present invention.

FIG. 18C is a diagram showing a syntax of slice header according toEmbodiment 1 of the present invention.

FIG. 19A is a diagram showing a syntax of coding unit (CU) according toEmbodiment 1 of the present invention.

FIG. 19B is a diagram showing a syntax of prediction unit (PU) accordingto Embodiment 1 of the present invention.

FIG. 19C is a diagram showing a syntax of transform unit (TU) accordingto Embodiment 1 of the present invention.

FIG. 20 is an overall configuration of a content providing system forimplementing content distribution services.

FIG. 21 shows an overall configuration of a digital broadcasting system.

FIG. 22 shows a block diagram illustrating an example of a configurationof a television.

FIG. 23 shows a block diagram illustrating an example of a configurationof an information reproducing/recording unit that reads and writesinformation from and on a recording medium that is an optical disk.

FIG. 24 shows an example of a configuration of a recording medium thatis an optical disk.

FIG. 25A shows an example of a cellular phone.

FIG. 25B is a block diagram showing an example of a configuration of acellular phone.

FIG. 26 illustrates a structure of multiplexed data.

FIG. 27 schematically shows how each stream is multiplexed inmultiplexed data.

FIG. 28 shows how a video stream is stored in a stream of PES packets inmore detail.

FIG. 29 shows a structure of TS packets and source packets in themultiplexed data.

FIG. 30 shows a data structure of a PMT.

FIG. 31 shows an internal structure of multiplexed data information.

FIG. 32 shows an internal structure of stream attribute information.

FIG. 33 shows steps for identifying video data.

FIG. 34 shows an example of a configuration of an integrated circuit forimplementing the moving picture coding method and the moving picturedecoding method according to each of embodiments.

FIG. 35 shows a configuration for switching between driving frequencies.

FIG. 36 shows steps for identifying video data and switching betweendriving frequencies.

FIG. 37 shows an example of a look-up table in which video datastandards are associated with driving frequencies.

FIG. 38A is a diagram showing an example of a configuration for sharinga module of a signal processing unit.

FIG. 38B is a diagram showing another example of a configuration forsharing a module of the signal processing unit.

DESCRIPTION OF EMBODIMENTS

Hereafter, the embodiments of the present invention will be describedwith reference to the drawings.

Embodiment 1

FIG. 2 is a block diagram showing the configuration of an image codingapparatus according to the present embodiment.

An image coding apparatus 1000 includes a coding processing unit 1100and a coding control unit 1200.

The coding processing unit 1100 generates a coded stream by codingmoving pictures on a block-by-block basis. The coding processing unit1100 includes a subtractor 1101, an orthogonal transform unit 1102, aquantization unit 1103, an entropy coding unit 1104, an inversequantization unit 1105, an inverse orthogonal transform unit 1106, anadder 1107, a deblocking filter 1108, a memory 1109, an intra predictionunit 1110, a motion compensation unit 1111, a motion estimation unit1112, and a switch 1113.

The subtractor 1101 obtains a moving picture and a prediction image fromthe switch 1113. The subtractor 1101 subtracts the prediction image fromthe current block to be coded included in the moving picture, togenerate a difference image.

The orthogonal transform unit 1102 performs orthogonal transform such asdiscrete cosine transform on the difference image generated by thesubtractor 1101, to transform the difference image into a coefficientblock comprising a plurality of frequency coefficients. The quantizationunit 1103 quantizes each of the frequency coefficients included in thecoefficient block, to generate a quantized coefficient block.

The entropy coding unit 1104 generates a coded stream by performingentropy coding (variable length coding) on the coefficient blockquantized by the quantization unit 1103 and a motion vector estimated bythe motion estimation unit 1112.

The inverse quantization unit 1105 performs inverse quantization on thecoefficient block quantized by the quantization unit 1103. The inverseorthogonal transform unit 1106 generates a decoded difference image byperforming inverse orthogonal transform such as inverse discrete cosinetransform on each of the frequency coefficients included in theinversely quantized coefficient block.

The adder 1107 generates a locally decoded image by obtaining aprediction image from the switch 1113 and by adding the prediction imageand the decoded difference image which is generated by the inverseorthogonal transform unit 1106.

The deblocking filter 1108 removes block distortion of the locallydecoded image generated by the adder 1107 and stores the locally decodedimage in the memory 1109.

The intra prediction unit 1110 generates a prediction image byperforming intra prediction on the current block to be coded using thelocally decoded image generated by the adder 1107.

The motion estimation unit 1112 estimates a motion vector for thecurrent block to be coded included in the moving picture, and outputsthe estimated motion vector to the motion compensation unit 1111 and theentropy coding unit 1104.

The motion compensation unit 1111 performs motion compensation on thecurrent block to be coded by referring to the image stored in the memory1109 as a reference image and by using the motion vector estimated bythe motion estimation unit 1112. The motion compensation unit 1111generates, by the motion compensation, a prediction image with respectto the current block to be coded.

When intra predictive coding is performed on the current block to becoded, the switch 1113 outputs the prediction image generated by theintra prediction unit 1110 to the subtractor 1101 and the adder 1107.When inter predictive coding is performed on the current block to becoded, the switch 1113 outputs the prediction image generated by themotion compensation unit 1111 to the subtractor 1101 and the adder 1107.

The coding control unit 1200 controls the coding processing unit 1100.More specifically, the coding control unit 1200 determines a processingunit in which a quantization parameter is stored and hierarchy depthinformation for specifying the location of the processing unit. Thequantization parameter is a parameter used for quantization by thequantization unit 1103 and inverse quantization by the inversequantization unit 1105. The processing units according to the presentembodiment are layered, and each of the processing units at any layercorresponds to the above described block. The hierarchy depthinformation is, for example, a parameter for specifying the layer havinga processing unit in which a quantization parameter is stored. Thecoding control unit 1200 instructs the entropy coding unit 1104 to storea quantization parameter in the above determined processing unit and tostore the hierarchy depth information in the header of the coded stream(for example, sequence header or picture header).

FIG. 3 is a block diagram showing the configuration of an image decodingapparatus according to the present embodiment.

An image decoding apparatus 2000 includes a decoding processing unit2100 and a decoding control unit 2200.

The decoding processing unit 2100 generates a decoded image by decodinga coded stream on a block-by-block basis. The decoding processing unit2100 includes an entropy decoding unit 2101, an inverse quantizationunit 2102, an inverse orthogonal transform unit 2103, an adder 2104, adeblocking filter 2105, a memory 2106, an intra prediction unit 2107, amotion compensation unit 2108, and a switch 2109.

The entropy decoding unit 2101 obtains a coded stream and performsentropy decoding (variable length decoding) on the coded stream.

The inverse quantization unit 2102 performs inverse quantization on thequantized coefficient block generated by entropy decoding by the entropydecoding unit 2101. The inverse orthogonal transform unit 2103 generatesa decoded difference image by performing inverse orthogonal transformsuch as inverse discrete cosine transform on each of the frequencycoefficients included in the inversely quantized coefficient block.

The adder 2104 generates a decoded image by obtaining a prediction imagefrom the switch 2109 and by adding the prediction image and the decodeddifference image which is generated by the inverse orthogonal transformunit 2103.

The deblocking filter 2105 removes block distortion of the decoded imagegenerated by the adder 2104, stores the decoded image in the memory2106, and outputs the decoded image.

The intra prediction unit 1110 generates a prediction image byperforming intra prediction on the current block to be decoded using thedecoded image generated by the adder 2104.

The motion compensation unit 2108 performs motion compensation on thecurrent block to be decoded by referring to the image stored in thememory 2106 as a reference image and by using the motion vectorgenerated by entropy decoding by the entropy decoding unit 2101. Themotion compensation unit 2108 generates, by the motion compensation, aprediction image with respect to the current block to be decoded.

When intra predictive coding is performed on the current block to bedecoded, the switch 2109 outputs the prediction image generated by theintra prediction unit 2107 to the adder 2104. When inter predictivecoding is performed on the current block to be decoded, the switch 2109outputs the prediction image generated by the motion compensation unit2108 to the adder 2104.

The decoding control unit 2200 controls the decoding processing unit2100. More specifically, the decoding control unit 2200 parses thehierarchy depth information stored in the header of the coded stream(for example, sequence header or picture header), based on the result ofthe entropy decoding by the entropy decoding unit 2101. The decodingcontrol unit 2200 identifies, based on the hierarchy depth information,the hierarchical layer having a processing unit in which a quantizationparameter is stored and parses the quantization parameter included inthe processing unit in the hierarchical layer. The decoding control unit2200 instructs the inverse quantization unit 2102 to perform inversequantization using the parsed quantization parameter.

FIG. 4 is an illustration diagram for describing layered processingunits (multi-hierarchical block structure).

The coding processing unit 1100 performs coding on the moving picturefor each of the processing units, and the decoding processing unit 2100decodes the coded stream for each of the processing units. Theprocessing unit is split into small processing units, and the smallprocessing units are layered such that each of the small processingunits is split into smaller processing units. It should be noted thatwhen the processing unit is smaller, the hierarchical layer in which theprocessing unit exists is deeper and lower and the value showing thehierarchical layer is greater. In contrast, when the processing unit isgreater, the hierarchical layer in which the processing unit exists isshallow and is in high level and the value showing the hierarchicallayer is small.

The processing unit includes a coding unit (CU), a prediction unit (PU),and a transform unit (TU). CU is a block of maximum 128×128 pixels, andcorresponds to the conventional macroblock. PU is a basic unit for interprediction. TU is a basic unit for orthogonal transform, and the size ofTU is the same as the size of the PU or is smaller than the size of thePU by one hierarchical layer. CU is, for example, divided into four subCUs, and one of the sub CUs includes the PU and the TU both having thesame size as the sub CU (in this case, PU and TU are overlapping witheach other). For example, the PU is further divided into four sub PUs,and TU is also further divided into four sub TUs. It should be notedthat when the processing unit is divided into small processing units,the small processing unit is a sub-processing unit. For example, whenthe processing unit is CU, the sub-processing unit is a sub CU. When theprocessing unit is PU, the sub-processing unit is a sub PU. When theprocessing unit is TU, the sub-processing unit is a sub TU.

The following will describe the detail.

Pictures are divided into slices. A slice is a sequence of largestcoding units. The location of the largest coding unit is specified bythe largest coding unit address IcuAddr.

Each coding unit including the largest coding unit is divided into fourcoding units recursively. It results to the quadtree segmentation of thecoding unit. The location of the coding unit is specified by the codingunit index cuIdx in which the upper-left sample (pixel or coefficient)of the largest coding unit is determined as an origin.

Once the coding unit is not allowed to be split, it is considered as theprediction unit. As similarly to the coding unit, the location of theprediction unit is specified by the prediction unit index puIdx in whichthe upper-left sample (pixel or coefficient) of the largest coding unitis determined as an origin.

The prediction unit may include multiple partitions (prediction unitpartitions or sub PUs). The prediction unit partition is specified bythe prediction partition index puPartIdx in which the upper-left sampleof the prediction unit is determined as an origin.

The prediction unit may include multiple transform units. As similarlyto the coding unit, the transform unit may be split into four smalltransform units (sub transform units) recursively. This allows quadtreesegmentation of the residual signal. The location of the transform unitis specified by the transform unit index tuIdx in which the upper-leftsample of the prediction unit is determined as an origin.

Here, the definitions of the processing units are as follows.

Coding tree block (CTB): A basic unit for specifying the quadtreesegmentation of the given square region. CTB has various sizes of asquare shape.

Largest coding tree block (LTCB): Coding tree block of the largest sizeallowed in the slice. A slice consists of non-overlapped LCTBs.

Smallest coding tree block (SCTB): Coding tree block of the smallestsize allowed in the slice. SCTB is not allowed to be split into smallerCTBs.

Prediction unit (PU): A basic unit for specifying the predictionprocess. The size of PU is the same as that of the CU which is notallowed to be split anymore. PU can be split into multiple partitionswhich may have arbitrary shapes whereas CU is allowed to be split intofour square shapes.

Transform unit (TU): A basic unit for specifying transform andquantization process.

Coding unit (CU): Same as coding tree block.

Largest coding unit (LCU): Same as largest coding tree block.

Smallest coding unit (SCU): Same as smallest coding tree block.

Moreover, the quantization parameters include one or more of thefollowing parameters: delta quantization scale parameter (delta QP or QPdelta), quantization offset parameter, an index (Qmatrix select idc),and quantization dead zone offset parameter. It should be noted that theindex is to select a quantization scale matrix from a plurality ofquantization scale matrixes.

The delta quantization scale parameter (delta QP or QP delta) is adifference between the quantization scale parameter to be applied totransform coefficients and the quantization scale parameter to bespecified by sequence header or slice header (or the previousquantization scale parameter in a Z-scan order).

The quantization offset parameter is also called quantization offset andis an adjustment value (offset value) for rounding a signal whenquantization is performed. Therefore, the image coding apparatus 1000codes the quantization offset when quantization is performed, and theimage decoding apparatus 2000 decodes the coded quantization offset.Then, the image decoding apparatus 2000 performs correction using thequantization offset when inverse quantization is performed on thetransform coefficients.

Index (Qmatrix select idc) is also called adaptive quantization matrixand is an index indicating which quantization scaling matrix is usedfrom a plurality of the quantization scaling matrixes. Moreover, Qmatrixselect idc shows whether or not the quantization scaling matrix is usedwhen there is only one quantization scaling matrix. It should be notedthat the adaptive quantization matrix can be controlled on ablock-by-block basis (processing unit).

The quantization dead zone offset parameter is also called adoptive deadzone, and is control information for adaptively changing the dead zoneon a block-by-block basis. The dead zone is a width in which thefrequency coefficients become zero by quantization (the previous widthwhich becomes plus 1 or minus 1 after quantization).

FIG. 5 is a diagram showing the configuration of a coded stream to begenerated by TMuC software.

In a coded stream generated by software of Test Model UnderConsideration (TMuC), delta QP is stored in LCU. In other words, in thecoded stream, the same quantization parameter such as delta QP isapplied to all coefficients included in LCU that is a large processingunit. As a result, the quantization parameter cannot be adjusted for adetail of the image, and image quality is decreased.

Therefore, in the coded stream generated by the image coding apparatus1000 and decoded by the image decoding apparatus 2000 according to thepresent embodiment, the quantization parameter is stored in a processingunit which is located at a lower hierarchical layer that is deeper thanLCU.

FIGS. 6A, 6B, and 6C each are a diagram showing the configuration of acoded stream according to the present embodiment.

As shown in FIG. 6A, in the coded stream according to the presentembodiment, LCU is split into four sub CUs, and delta QP to be appliedto each of the sub CUs is stored in the sub CU. In other words, when LCUis the first hierarchical layer, delta QP is stored in CU which islocated lower by two hierarchical layers from LCU. Moreover, in the subCU, delta QP is disposed after all transform coefficients included inthe sub CU.

Furthermore, in the coded stream according to the present embodiment,hierarchy depth information which indicates the lowest hierarchicallayer of the processing unit (Max_quantization_unit_hierarchy_depth) inwhich delta QP is stored is stored in the sequence header. For example,

Max_quantization_unit_hierarchy_depth=2.

The image coding apparatus 1000 generates and outputs the coded stream.Meanwhile, the image decoding apparatus 2000 identifies the processingunit in which delta QP is stored (sub CU located at the secondhierarchical layer) by parsing the hierarchy depth information(Max_quantization_unit_hierarchy_depth) stored in the sequence header ofthe coded stream, and parses the delta QP stored in the processing unit.Then, the image decoding apparatus 2000 performs inverse quantization onthe delta QP by applying the delta QP to each of the transformcoefficients of the sub CU which stores the delta QP.

As shown in FIG. 6B, Qmatrix select idc may be stored instead of deltaQP. Furthermore, as shown in FIG. 6C, the quantization parameterincluding delta QP and Qmatrix select idc may be stored.

FIG. 7 is a diagram showing the configuration of another coded streamaccording to the present embodiment.

In the coded stream shown in FIGS. 6A to 6C, the quantization parameteris stored in the sub CU located at the second hierarchical layer.However, as shown in FIG. 7, the quantization parameter may be stored inthe sub CU or the sub TU located at the deeper third hierarchical layer(Max_quantization_unit_hierarchy_depth=3).

FIGS. 8A and 8B each are a diagram showing the configuration of a stillanother coded stream according to the present embodiment.

In the coded stream shown in FIG. 8A, the delta QP to be applied to TUor sub TU is stored in the TU or the sub TU. In the TU or the sub TU,the delta QP is disposed after all the transform coefficients includedin the TU or the sub TU.

Moreover, as shown in FIG. 8B, the quantization parameter other thandelta QP and the quantization parameter including delta QP and Qmatrixselect idc may be stored as a quantization unit.

FIGS. 9A and 9B each are a diagram showing the storage location ofMax_quantization_unit_hierarchy_depth.

Max_quantization_unit_hierarchy_depth D300 is stored in the sequenceheader as shown in FIG. 9A. Moreover,Max_quantization_unit_hierarchy_depth D302 is stored in the pictureheader as shown in FIG. 9B. In other words, the image coding apparatus1000 writes hierarchy depth information into the picture header for apicture comprising a plurality of processing units. As a result,hierarchy depth information is stored in the picture header.

FIGS. 10A to 10D are each a diagram for describing types of quantizationparameters.

Quantization parameter or quantization unit D600, as shown in FIGS. 10Ato 10D, includes at least one of delta quantization scale parameterD602, quantization dead zone offset parameter D604, index D606, andquantization offset parameter D608. It should be noted that deltaquantization scale parameter D602 is delta QP, and index D606 is Qmatrixselect idc (adaptive quantization matrix).

FIG. 11 is a flowchart showing decoding of delta QP by the imagingdecoding apparatus 2000.

First, the image decoding apparatus 2000 decodes hierarchy depthinformation (Max_quantization_unit_hierarchy_depth) stored in the header(Step 51), and determines the smallest size of the quantizationprocessing unit (minimum quantization unit) (Step S2). Next, the imagedecoding apparatus 2000 determines whether or not the current CU to bedecoded has the size (Step S3). Here, when it is determined that thecurrent CU to be decoded has the size of minimum quantization unit (Yesin Step S3), the image decoding apparatus 2000 decodes delta QP storedin the CU (Step S4). Meanwhile, when it is determined that the currentCU to be decoded does not have the size of minimum quantization unit (Noin Step S3), the image decoding apparatus 2000 further determineswhether or not the flag of the current CU to be decoded(split_coding_unit flag) is zero and the size of the current CU to bedecoded is larger than the size of minimum quantization unit (Step S5).It should be noted that when the above described split_coding_unit flagis zero, this shows that the flag cannot be further split. When theabove described split_coding_unit flag is one, this shows that the flagcan split the CU further. In other words, the image decoding apparatus2000, in Step S5, determines whether or not the current CU to be decodedcannot be further split and the current CU to be decoded is locatedhigher than the hierarchical layer indicated by the hierarchy depthinformation. Here, it is determined that the flag is zero and the sizeof the current CU to be decoded is large (Yes in Step S5), the imagedecoding apparatus 2000 decodes delta QP stored in the current CU to bedecoded (Step S6).

FIG. 12 is a flowchart showing computation of delta QP (quantizationscale parameter) by the imaging decoding apparatus 2000.

First, the image decoding apparatus 2000 determines, by summingcoded_block_flag (CBF) in each TU in the quartered processing unit,whether or not the TU for luminance and chrominance included in thecurrent TU to be decoded is coded (Steps S11 and S12). It should benoted that each of the TUs stores the above described coded_block_flagthat is a flag showing whether or not it is a transform coefficient.Here, when it is determined the TU is coded (Yes in Step S12), the imagedecoding apparatus 2000 decodes delta QP included in the TU (Step S14).Meanwhile, when it is determined that the TU is not coded (No in StepS12), the image decoding apparatus 2000 sets delta QP to zero (StepS13). Furthermore, the image decoding apparatus 2000 determines QP ofthe previous CU in a Z-scan order (Step S15), and computes QP of thecurrent CU to be decoded (Step S16).

As described above, when the processing units are hierarchicallylayered, the image coding apparatus 1000 according to the presentembodiment can write a different parameter (for example, quantizationparameter) which is different for each of the small processing unitsthat are located at the lower hierarchical layers. As a result, theimage decoding apparatus 2000 can decode the processing units byapplying a different parameter to each of the small processing units,and therefore image quality can be improved. Moreover, by writing, intoa header, the hierarchy depth information for specifying thehierarchical layer having a processing unit in which a parameternecessary for decoding is stored, the hierarchical layer can be notifiedto the image decoding apparatus 2000. Therefore, the hierarchical layercan be set at an arbitrary hierarchical layer without limiting thehierarchical layer to the lowest hierarchical layer. Therefore, theamount of coding for all parameters included in a coded stream can bereduced compared with the case where a parameter is stored for each ofthe smallest processing unit located at the lowest hierarchical layer,and coding efficiency can be improved. With this, image quality andcoding efficiency can be improved simultaneously. Moreover, since theimage decoding apparatus 2000 identifies, by parsing the hierarchy depthinformation, a hierarchical layer having a processing unit in which aparameter is stored is specified, it is possible for the image decodingapparatus 2000 to reduce the burden of a process of searching theprocessing unit in which the parameter is stored and to appropriatelydecode the coded stream generated by the image coding apparatus 1000. Itshould be noted that in the present embodiment, quantization parameteris cited as an example of parameter. However, any form of parameter isacceptable.

Modification 1

FIG. 13 is a flowchart of decoding by an image decoding apparatus 2000according to Modification 1 of the present embodiment.

The image decoding apparatus 2000 first parses hierarchy depthinformation (Max_quantization_unit_hierachy_depth) stored in the pictureheader (Step S7000), and parses the flag of CU (Step S702). Next, theimage decoding apparatus 2000, based on the parsed flag, splits the CUinto smaller sub CUs (Step S704). Then, the image decoding apparatus2000 determines the hierarchical layer of the sub CU (Step S706) anddetermines whether or not the determined hierarchical layer matches thehierarchical layer indicated by Max_quantization_unit_hierarchy_depth(Step S708).

When it is determined that the determined layer matches the hierarchicallayer specified by Max_quantization_unit_hierarchy_depth (Yes in StepS708), the image decoding apparatus 2000 parses quantization parameterstored in the sub CU (Step S710) and decodes the sub CU by performinginverse quantization with the parsed quantization parameter (Step S712).

Meanwhile, when it is determined in Step S708 that the determined layerdoes not match the hierarchical layer specified byMax_quantization_unit_hierarchy_depth (No in Step S708), the imagedecoding apparatus 2000 determines whether the sub CU cannot be furthersplit into four smaller sub CUs, based on the above described flag (StepS714). Here, when it is determined that the sub CU cannot be furthersplit (Yes in Step S714), the image decoding apparatus 2000 performs theprocesses of the above described Steps S710 and S712 on the sub CU.Meanwhile, when it is determined that the sub CU can be further split(No in Step S714), the image decoding apparatus 2000 selects any one ofthe four smaller sub CUs (Step S716), and performs the processes fromS706 on the selected sub CU.

FIG. 14 is a flowchart of coding by an image coding apparatus 1000according to Modification 1 of the present embodiment.

First, the image coding apparatus 1000 writes hierarchy depthinformation (Max_quantization_unit_hierarchy_depth) into the pictureheader (Step S800), and determines the most appropriate size forsplitting the CU (Step S802). Next, the image coding apparatus 1000writes, into the CU, flag for splitting the CU into processing units ofthe determined size (Step S804). Then, the image coding apparatus 1000determines the hierarchical layer of the processing unit to be coded (CUor sub CU) (Step S808), and determines whether or not the determinedhierarchical layer matches the hierarchical layer indicated byMax_quantization_unit_hierarchy_depth that is previously written (StepS808).

When it is determined that they match with each other (Yes in StepS808), the image coding apparatus 1000 writes quantization parameterinto the processing unit (CU or sub CU) (Step S810), the image codingapparatus 1000 codes the processing unit by performing quantizationusing the written quantization parameter (Step S812). Furthermore, theimage coding apparatus 1000 performs inverse quantization using thewritten quantization parameter to decode the coded processing unit (StepS814).

Meanwhile, when it is determined in Step S808 that they do not matchwith each other (No in Step S808), the image coding apparatus 1000determines whether the processing unit cannot be further split into foursmaller sub CUs, based on the above described flag (Step S816). Here,when it is determined that the sub CU cannot be further split (Yes inStep S816), the image coding apparatus 1000 performs the above describedsteps starting from Step S810 on the processing unit. Meanwhile, when itis determined that the sub CU can be further split (No in Step S816),the image coding apparatus 1000 selects any one of the four smaller subCUs (Step S818), and performs the processes from S806 on the selectedsub CU.

Modification 2

FIGS. 15A and 15B are each a flowchart of decoding by an image decodingapparatus 2000 according to Modification 2 of the present embodiment.

The image decoding apparatus 2000 first parses hierarchy depthinformation (Max_quantization_unit_hierachy_depth) stored in the pictureheader (Step S900), and parses the flag of CU (Step S902). Next, theimage decoding apparatus 2000, based on the parsed flags, splits the CUinto smaller sub CUs (Step S904). Then, the image decoding apparatus2000 determines the hierarchical layer of the sub CU (Step S906) anddetermines whether or not the determined hierarchical layer matches thehierarchical layer indicated by Max_quantization_unit_hierarchy_depth(Step S908).

When it is determined that they match with each other (Yes in StepS908), the image decoding apparatus 2000 parses the quantizationparameter stored in the sub CU (processing unit) (Step S910) and decodesthe sub CU by performing inverse quantization with the parsedquantization parameter (Step S912).

Meanwhile, when it is determined in Step S908 that they do not matchwith each other (No in Step S908), the image decoding apparatus 2000determines whether the sub CU cannot be further split into four smallersub CUs, based on the above described flag (Step S914). When it isdetermined that the sub CU can be further split (No in Step S914), theimage decoding apparatus 2000 selects any one of the four smaller subCUs (Step S928) and performs the processes from S906 on the selected subCU.

Meanwhile, when it is determined in Step 914 that the sub CU cannot befurther split (Yes in Step S914), the image decoding apparatus 2000parses transform split flag located within the TU of the sub CU (StepS916), and splits the TU into sub TUs that are smaller processing units,based on the parsed transform split flag (Step S918). Furthermore, theimage decoding apparatus 2000 determines the hierarchical layer from LCUwith respect to the sub TU (Step S920) and determines whether or not thedetermined hierarchical layer matches the hierarchical layer indicatedby Max_quantization_unit_hierarchy_depth (Step S922).

When it is determined that they match with each other (Yes in StepS922), the image decoding apparatus 2000 performs the processes fromS910 on the sub TU. Meanwhile, when it is determined in Step S922 thatthey do not match with each other (No in Step S922), the image decodingapparatus 2000 determines whether the sub TU cannot be further splitinto four smaller sub TUs, based on the above described transform splitflag (Step S926). When it is determined that the sub TU can be furthersplit (No in Step S926), the image decoding apparatus 2000 selects anyone of the four smaller sub TUs (Step S924) and performs the processesfrom S920 on the selected sub TU. When it is determined that the sub TUcannot be further split (Yes in Step S926), the image decoding apparatus2000 performs the processes from S910.

FIGS. 16A and 16B are each a flowchart of coding by an image codingapparatus 1000 according to Modification 2 of the present embodiment.

First, the image coding apparatus 1000 writes hierarchy depthinformation (Max_quantization_unit_hierarchy_depth) into the pictureheader (Step S1000), and determines the most appropriate size forsplitting the CU (Step S1002). Next, the image coding apparatus 1000writes, into the CU, flag for splitting the CU into processing units ofthe determined sizes (Step S1004). Then, the image coding apparatus 1000determines the hierarchical layer of the processing unit to be coded (CUor sub CU) (Step S1006), and determines whether or not the determinedhierarchical layer matches the hierarchical layer indicated byMax_quantization_unit_hierarchy_depth that is previously written (StepS1008).

When it is determined that they match with each other (Yes in StepS1008), the image coding apparatus 1000 writes quantization parameterinto the processing unit (CU or sub CU) (Step S1010), the image codingapparatus 1000 codes the processing unit by performing quantizationusing the written quantization parameters (Step S1030). Furthermore, theimage coding apparatus 1000 performs inverse quantization using thewritten quantization parameter to decode the coded processing unit (StepS1012).

Meanwhile, when it is determined in Step S1008 that they do not matchwith each other (No in Step S1008), the image coding apparatus 1000determines whether the processing unit cannot be further split into foursmaller sub CUs, based on the above described flag (Step S1014). When itis determined that the sub CU can be further split (No in Step S1014),the image coding apparatus 1000 selects any one of the four smaller subCUs (Step S1028) and performs the processes from S1006 on the selectedsub CU.

Meanwhile, when it is determined that the sub CU cannot be further splitin Step S1014 (Yes in Step S1014), the image coding apparatus 1000determines the most appropriate size for splitting the TU within theprocessing unit (CU or sub CU) (Step S1016), and writes, into the TU,flag (transform split flag) for splitting the TU into processing unitsof the determined sizes (Step S1018). Next, the image coding apparatus1000 determines the hierarchical layer from LCU with respect to theprocessing unit to be coded (TU or sub TU) (Step S1020), and determineswhether or not the determined hierarchical layer matches thehierarchical layer indicated by Max_quantization_unit_hierarchy_depththat is previously written (Step S1022).

When it is determined that they match with each other (Yes in StepS1022), the image coding apparatus 1000 performs the processes from StepS1010 on the processing unit (TU or sub TU) (Step S1010). Meanwhile,when it is determined in Step S1022 that they do not match with eachother (No in Step S1022), the image coding apparatus 1000 determineswhether the processing unit (TU or sub TU) cannot be further split intofour smaller sub TUs, based on the above described transform split flag(Step S1026). When it is determined that the sub TU can be further split(No in Step S1026), the image coding apparatus 1000 selects any one ofthe four smaller sub TUs (Step S1024) and performs the processes fromS1020 on the selected sub TU. When it is determined in Step S1026 thatthe sub TU cannot be further split (Yes in Step S1026), the image codingapparatus 1000 performs the processes from S1010. In other words, theimage coding apparatus 1000 writes quantization parameter into theprocessing unit (TU or sub TU) (Step S1010), the image coding apparatus1000 codes the processing unit by performing quantization using thewritten quantization parameters (Step S1030). Furthermore, the imagecoding apparatus 1000 performs inverse quantization using the writtenquantization parameter to decode the coded processing unit (Step S1012).

The problems and the solution in the present invention are as follows.

In other words, by splitting a picture into large coding units, codingefficiency can be improved. However, when the quantization parameter isset to a large coding unit, flexibility in adjusting the size of thepicture is lost in the image coding apparatus since the size of thecoding unit is large. The quantization parameter includes at least oneof quantization scale parameter, quantization offset parameter, andindex. It should be noted that the index is to select a quantizationscale matrix from among a plurality of quantization scale matrixes.

For example, an important feature of coding and decoding of a movingpicture is that video device requiring low delay in teleconference andsecurity camera can adjust the maximum size of a picture. With this, itis necessary for the quantization parameter to be adjusted with thesmallest unit of a picture. Meanwhile, the other video devices do notrequire the above described feature, and can improve coding efficiencyby reducing an overhead for transmitting the quantization parameters.

Here, the coding unit, the prediction unit, and the transform unit arebasic units of the High Efficiency Video Coding (HEVC) standard. QP thatis a quantization scale parameter is a parameter used for inversescaling process on a difference value (delta value), and is transmittedon a coding unit level. In Test Model Under Consideration (TMuC) ofHEVC, the delta quantization scale parameter is not transmitted.However, in software, the delta quantization scale parameter istransmitted to the end of the quantization of the largest coding unit.However, when PU that is the prediction unit is skipped, the depth ofthe CU that is the coding unit is zero. This means that Y block, Ublock, and V block are not coded.

In other words, there are two problems (Problems 1 and 2) as follows.

Problem 1: the coding delta quantization scale parameter is restrictedonly on a largest coding unit level. It may be difficult for videodevice having low delay or constant bit rate to adjust a bit for each ofthe coding units. In other words, in TMuC standard and TMuC software,the restriction is strict on the storage position of information, andthe quantization parameter can be transmitted only with the largest CU.As a result, it is not possible for the quantization parameter to becontrolled by a smaller unit (processing unit).

Problem 2: when TU that is a transform unit is not coded, thequantization parameter is not necessary. However, the current techniquechecks when TU and PU are skipped. Since TU and PU are separated, thetransmission of QP delta only depends on TU. Moreover, when there are notransform coefficients (coefficients generated by quantization andorthogonal transform of an image in a space region), it is necessary foran unnecessary quantization parameter with respect to the transformcoefficients to be transmitted. As a result, a coded stream which showsa coded image becomes redundant.

In order to solve the above described problems, a new method is providedfor transmitting quantization parameter for each maximum coding unit.The transmission method allows the image coding apparatus to select alevel for a quantization parameter included in the coding unit to betransmitted in order to ensure both the functionality of fine bitcontrol of a block and high coding efficiency.

What is novel about the present invention is high flexibility orfunctionality for the image coding device in which the location of thequantization parameter in the largest coding unit of a picture isdetermined for better control of data rate. The functionality is notpresent in any prior art and can help improve image quality of a codedmoving picture by combining uses of the largest coding unit and thequantization parameter. What is also novel about the present inventionis the location of the quantization parameter in the coding unit.Especially, in the conventional technique, the quantization parameter isincluded in the header of the coding unit such as macroblock. However,in the present invention, after the prediction and differenceinformation on the block is coded, the quantization parameter is codedat the end of the coding unit.

In other words, there are solutions to the above described Problems 1and 2 (Solution 1 and Solution 2) as follows.

Solution 1: In order to transmit delta QP at a small CU level, hierarchydepth information is inserted into header (sequence parameterset/picture parameter set/slice header). In other words, the imagecoding apparatus stores the quantization parameter in a small unit(processing unit) located deeper than the maximum CU, and stores, in aheader such as sequence header or picture header, hierarchy depthinformation for specifying the hierarchical layer (depth of thehierarchical layer) in which the processing unit exists. The imagedecoding apparatus specifies the hierarchical layer by parsing thehierarchy depth information (depth of the hierarchical layer) in theheader, and parses the quantization parameter stored in the processingunit located in the specified hierarchical layer. Here, the hierarchydepth information may indicate the deepest (lowest located) hierarchicallayer in which the processing unit storing the quantization parametercan exist. In this case, the image decoding apparatus specifies thelowest hierarchical layer indicated by the hierarchy depth informationor the hierarchical layer which is located higher than the hierarchicallayer and is other than the highest hierarchical layer. Moreover, thehierarchy depth information may be a flag which shows whether or not thequantization parameter is stored in a CU of a predetermined hierarchicallayer (for example, the CU which is located at the lowest hierarchicallayer).

Solution 2: In order to skip the transmission of delta QP, a newcondition is introduced for checking TU coded block flag or a pattern.Moreover, the image coding apparatus dispose the quantization parameterat the end of the TU when transmitting the quantization parameter. Withthis, the image decoding apparatus can determine when the quantizationparameter is not necessary (when there are no transform coefficients).As a result, the image coding apparatus does not have to transmitunnecessary quantization parameters, and the amount of coding can bereduced.

As described above, the image decoding method and the image codingmethod according to the present invention have been described withreference to the above described embodiments and modifications. However,the present invention is not defined only by these.

For example, the processes such as Steps S3 and S5 in FIG. 11 areincluded in the image decoding method according to Embodiment 1 and itsmodification. However, the prevent invention can generate the abovedescribed effect without the processes.

FIG. 17A is a flowchart showing an image decoding method according tothe present invention.

The image decoding method according to the present invention is an imagedecoding method for decoding a coded stream including a plurality ofprocessing units and a header with respect to the processing units, thecoded stream being generated by coding a moving picture. Here, at leastone of the processing units is layered such that the processing unit isdivided into a plurality of smaller processing units. In the imagedecoding method, first, by parsing hierarchy depth information stored inthe header, a hierarchical layer is specified in which the processingunit storing a parameter necessary for decoding exists (Step S101).Next, by using the parameter stored in the processing unit located atthe specified hierarchical layer, the processing unit is decoded (StepS102).

By performing the processes of Steps S101 and S102, it is possible toobtain the same effect as that obtained from Embodiment 1. The otherprocesses are not essential for the present invention. Moreover, theimage decoding apparatus according to the present invention can obtainthe same effect as that obtained from Embodiment 1 by including aconstituent element which performs each of the processes of Step S101and Step S102. The other constituent elements are not essential for thepresent invention. It should be noted that in the image decodingapparatus 2000 according to Embodiment 1, the decoding control unit 2200performs the process of Step S101 and the decoding processing unit 2100performs the process of Step S102.

Moreover, the processes such as Step S804 in FIG. 14 are included in theimage coding method according to Embodiment 1 and its modification.However, the prevent invention can generate the above described effectwithout the processes.

FIG. 17B is a flowchart showing an image coding method according to thepresent invention.

The image decoding method according to the present invention is an imagedecoding method for generating, by coding a moving picture, a codedstream including a plurality of processing units and a header withrespect to the processing units. Here, at least one of the processingunits is layered such that the processing unit is split into a pluralityof smaller processing units. In the image coding method, a movingpicture is first coded (Step S111). Next, hierarchy depth informationfor specifying the hierarchical layer having a processing unit in whicha parameter necessary for decoding is stored is written into the header(Step S112). Furthermore, the parameter is written into the processingunit located in the hierarchical layer specified by the hierarchy depthinformation (Step S113).

By performing the processes of Steps S111 and S113, it is possible toobtain the same effect as that obtained from Embodiment 1. The otherprocesses are not requisite for the present invention. Moreover, theimage decoding apparatus according to the present invention can obtainthe same effect as that obtained from Embodiment 1 by including aprocessing unit which performs each of the processes of Step S111 toStep S113. The other constituent elements are not essential for thepresent invention. It should be noted that in the image coding apparatus1000 according to Embodiment 1, the entropy coding unit 1104 performsthe processes of Steps S111 to S113, based on the control by the codingcontrol unit 1200.

It should be noted a syntax of the header related to the presentinvention are shown in FIGS. 18A to 18C. The syntaxes of the processingunits related to the present invention (CU, PU, and TU) are shown inFIGS. 19A to 19C.

FIG. 18A is a diagram showing the syntax of the sequence header. In thesequence header, for example, the number of maximum reference framesthat can be referred (max_num_ref_frames) and the size of the picture(pic_width_in_luma_samples, pic_height_in_luma_samples) are defined.

FIG. 18B is a diagram showing the syntax of the picture header. In thepicture header, as shown in part d1 of the syntax, the number ofreference indexes to be held for each reference direction (forwarddirection and backward direction) is defined, and an initial QP (numberobtained by subtracting 26 from the initial QP) is defined.

FIG. 18C is a diagram showing the syntax of the slice header. The sliceheader, as shown in part d2 of the syntax, is configured such that thenumber of the above described reference indexes to be held can berewritten for each slice. Moreover, the slice header, as shown inanother part d3 of the syntax, defines the difference value of QP fromthe initial QP which is defined by the above described picture header.

FIG. 19A is a diagram showing the syntax of CU. In the CU, as shown inparts d4 and d5 of the syntax, PU and TU with respect to the CU aredefined.

FIG. 19B is a diagram showing the syntax of PU. The PU has, as shown inparts d6 and d8 of the syntax, a reference index for each referencedirection, and has, as shown in other parts d7 and d9 of the syntax,adaptive motion vector resolution switch flag (mvres) for each referencedirection.

FIG. 19C is a diagram showing the syntax of TU. The TU has, as shown inpart d10 of the syntax, coefficients (transform coefficients) in whichorthogonal transform and quantization are performed on the differenceimage.

Embodiment 2

The processing described in each of embodiments can be simplyimplemented in an independent computer system, by recording, in arecording medium, a program for implementing the configurations of themoving picture coding method (image coding method) and the movingpicture decoding method (image decoding method) described in each ofembodiments. The recording media may be any recording media as long asthe program can be recorded, such as a magnetic disk, an optical disk, amagnetic optical disk, an IC card, and a semiconductor memory.

Hereinafter, the applications to the moving picture coding method (imagecoding method) and the moving picture decoding method (image decodingmethod) described in each of embodiments and systems using thereof willbe described.

FIG. 20 illustrates an overall configuration of a content providingsystem ex100 for implementing content distribution services. The areafor providing communication services is divided into cells of desiredsize, and base stations ex106, ex107, ex108, ex109, and ex110 which arefixed wireless stations are placed in each of the cells.

The content providing system ex100 is connected to devices, such as acomputer ex111, a personal digital assistant (PDA) ex112, a cameraex113, a cellular phone ex114 and a game machine ex115, via the Internetex101, an Internet service provider ex102, a telephone network ex104, aswell as the base stations ex106 to ex110, respectively.

However, the configuration of the content providing system ex100 is notlimited to the configuration shown in FIG. 20, and a combination inwhich any of the elements are connected is acceptable. In addition, eachdevice may be directly connected to the telephone network ex104, ratherthan via the base stations ex106 to ex110 which are the fixed wirelessstations. Furthermore, the devices may be interconnected to each othervia a short distance wireless communication and others.

The camera ex113, such as a digital video camera, is capable ofcapturing video. A camera ex116, such as a digital camera, is capable ofcapturing both still images and video. Furthermore, the cellular phoneex114 may be the one that meets any of the standards such as GlobalSystem for Mobile Communications (GSM) (registered trademark), CodeDivision Multiple Access (CDMA), Wideband-Code Division Multiple Access(W-CDMA), Long Term Evolution (LTE), and High Speed Packet Access(HSPA). Alternatively, the cellular phone ex114 may be a PersonalHandyphone System (PHS).

In the content providing system ex100, a streaming server ex103 isconnected to the camera ex113 and others via the telephone network ex104and the base station ex109, which enables distribution of images of alive show and others. In such a distribution, a content (for example,video of a music live show) captured by the user using the camera ex113is coded as described above in each of embodiments (i.e., the camerafunctions as the image coding apparatus according to an aspect of thepresent invention), and the coded content is transmitted to thestreaming server ex103. On the other hand, the streaming server ex103carries out stream distribution of the transmitted content data to theclients upon their requests. The clients include the computer ex111, thePDA ex112, the camera ex113, the cellular phone ex114, and the gamemachine ex115 that are capable of decoding the above-mentioned codeddata. Each of the devices that have received the distributed datadecodes and reproduces the coded data (i.e., functions as the imagedecoding apparatus according to an aspect of the present invention).

The captured data may be coded by the camera ex113 or the streamingserver ex103 that transmits the data, or the coding processes may beshared between the camera ex113 and the streaming server ex103.Similarly, the distributed data may be decoded by the clients or thestreaming server ex103, or the decoding processes may be shared betweenthe clients and the streaming server ex103. Furthermore, the data of thestill images and video captured by not only the camera ex113 but alsothe camera ex116 may be transmitted to the streaming server ex103through the computer ex111. The coding processes may be performed by thecamera ex116, the computer ex111, or the streaming server ex103, orshared among them.

Furthermore, the coding and decoding processes may be performed by anLSI ex500 generally included in each of the computer ex111 and thedevices. The LSI ex500 may be configured of a single chip or a pluralityof chips. Software for coding and decoding video may be integrated intosome type of a recording medium (such as a CD-ROM, a flexible disk, anda hard disk) that is readable by the computer ex111 and others, and thecoding and decoding processes may be performed using the software.Furthermore, when the cellular phone ex114 is equipped with a camera,the video data obtained by the camera may be transmitted. The video datais data coded by the LSI ex500 included in the cellular phone ex114.

Furthermore, the streaming server ex103 may be composed of servers andcomputers, and may decentralize data and process the decentralized data,record, or distribute data.

As described above, the clients may receive and reproduce the coded datain the content providing system ex100. In other words, the clients canreceive and decode information transmitted by the user, and reproducethe decoded data in real time in the content providing system ex100, sothat the user who does not have any particular right and equipment canimplement personal broadcasting.

Aside from the example of the content providing system ex100, at leastone of the moving picture coding apparatus (image coding apparatus) andthe moving picture decoding apparatus (image decoding apparatus)described in each of embodiments may be implemented in a digitalbroadcasting system ex200 illustrated in FIG. 21. More specifically, abroadcast station ex201 communicates or transmits, via radio waves to abroadcast satellite ex202, multiplexed data obtained by multiplexingaudio data and others onto video data. The video data is data coded bythe moving picture coding method described in each of embodiments (i.e.,data coded by the image coding apparatus according to an aspect of thepresent invention). Upon receipt of the multiplexed data, the broadcastsatellite ex202 transmits radio waves for broadcasting. Then, a home-useantenna ex204 with a satellite broadcast reception function receives theradio waves. Next, a device such as a television (receiver) ex300 and aset top box (STB) ex217 decodes the received multiplexed data, andreproduces the decoded data (i.e., functions as the image decodingapparatus according to an aspect of the present invention).

Furthermore, a reader/recorder ex218 (i) reads and decodes themultiplexed data recorded on a recording medium ex215, such as a DVD anda BD, or (i) codes video signals in the recording medium ex215, and insome cases, writes data obtained by multiplexing an audio signal on thecoded data. The reader/recorder ex218 can include the moving picturedecoding apparatus or the moving picture coding apparatus as shown ineach of embodiments. In this case, the reproduced video signals aredisplayed on the monitor ex219, and can be reproduced by another deviceor system using the recording medium ex215 on which the multiplexed datais recorded. It is also possible to implement the moving picturedecoding apparatus in the set top box ex217 connected to the cable ex203for a cable television or to the antenna ex204 for satellite and/orterrestrial broadcasting, so as to display the video signals on themonitor ex219 of the television ex300. The moving picture decodingapparatus may be implemented not in the set top box but in thetelevision ex300.

FIG. 22 illustrates the television (receiver) ex300 that uses the movingpicture coding method and the moving picture decoding method describedin each of embodiments. The television ex300 includes: a tuner ex301that obtains or provides multiplexed data obtained by multiplexing audiodata onto video data, through the antenna ex204 or the cable ex203, etc.that receives a broadcast; a modulation/demodulation unit ex302 thatdemodulates the received multiplexed data or modulates data intomultiplexed data to be supplied outside; and amultiplexing/demultiplexing unit ex303 that demultiplexes the modulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data coded by a signal processing unit ex306 into data.

The television ex300 further includes: a signal processing unit ex306including an audio signal processing unit ex304 and a video signalprocessing unit ex305 that decode audio data and video data and codeaudio data and video data, respectively (which function as the imagecoding apparatus and the image decoding apparatus according to theaspects of the present invention); and an output unit ex309 including aspeaker ex307 that provides the decoded audio signal, and a display unitex308 that displays the decoded video signal, such as a display.Furthermore, the television ex300 includes an interface unit ex317including an operation input unit ex312 that receives an input of a useroperation. Furthermore, the television ex300 includes a control unitex310 that controls overall each constituent element of the televisionex300, and a power supply circuit unit ex311 that supplies power to eachof the elements. Other than the operation input unit ex312, theinterface unit ex317 may include: a bridge ex313 that is connected to anexternal device, such as the reader/recorder ex218; a slot unit ex314for enabling attachment of the recording medium ex216, such as an SDcard; a driver ex315 to be connected to an external recording medium,such as a hard disk; and a modem ex316 to be connected to a telephonenetwork. Here, the recording medium ex216 can electrically recordinformation using a non-volatile/volatile semiconductor memory elementfor storage. The constituent elements of the television ex300 areconnected to each other through a synchronous bus.

First, the configuration in which the television ex300 decodesmultiplexed data obtained from outside through the antenna ex204 andothers and reproduces the decoded data will be described. In thetelevision ex300, upon a user operation through a remote controllerex220 and others, the multiplexing/demultiplexing unit ex303demultiplexes the multiplexed data demodulated by themodulation/demodulation unit ex302, under control of the control unitex310 including a CPU. Furthermore, the audio signal processing unitex304 decodes the demultiplexed audio data, and the video signalprocessing unit ex305 decodes the demultiplexed video data, using thedecoding method described in each of embodiments, in the televisionex300. The output unit ex309 provides the decoded video signal and audiosignal outside, respectively. When the output unit ex309 provides thevideo signal and the audio signal, the signals may be temporarily storedin buffers ex318 and ex319, and others so that the signals arereproduced in synchronization with each other. Furthermore, thetelevision ex300 may read multiplexed data not through a broadcast andothers but from the recording media ex215 and ex216, such as a magneticdisk, an optical disk, and a SD card. Next, a configuration in which thetelevision ex300 codes an audio signal and a video signal, and transmitsthe data outside or writes the data on a recording medium will bedescribed. In the television ex300, upon a user operation through theremote controller ex220 and others, the audio signal processing unitex304 codes an audio signal, and the video signal processing unit ex305codes a video signal, under control of the control unit ex310 using thecoding method described in each of embodiments. Themultiplexing/demultiplexing unit ex303 multiplexes the coded videosignal and audio signal, and provides the resulting signal outside. Whenthe multiplexing/demultiplexing unit ex303 multiplexes the video signaland the audio signal, the signals may be temporarily stored in thebuffers ex320 and ex321, and others so that the signals are reproducedin synchronization with each other. Here, the buffers ex318, ex319,ex320, and ex321 may be plural as illustrated, or at least one buffermay be shared in the television ex300. Furthermore, data may be storedin a buffer so that the system overflow and underflow may be avoidedbetween the modulation/demodulation unit ex302 and themultiplexing/demultiplexing unit ex303, for example.

Furthermore, the television ex300 may include a configuration forreceiving an AV input from a microphone or a camera other than theconfiguration for obtaining audio and video data from a broadcast or arecording medium, and may code the obtained data. Although thetelevision ex300 can code, multiplex, and provide outside data in thedescription, it may be capable of only receiving, decoding, andproviding outside data but not the coding, multiplexing, and providingoutside data.

Furthermore, when the reader/recorder ex218 reads or writes multiplexeddata from or on a recording medium, one of the television ex300 and thereader/recorder ex218 may decode or code the multiplexed data, and thetelevision ex300 and the reader/recorder ex218 may share the decoding orcoding.

As an example, FIG. 23 illustrates a configuration of an informationreproducing/recording unit ex400 when data is read or written from or onan optical disk. The information reproducing/recording unit ex400includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406,and ex407 to be described hereinafter. The optical head ex401 irradiatesa laser spot in a recording surface of the recording medium ex215 thatis an optical disk to write information, and detects reflected lightfrom the recording surface of the recording medium ex215 to read theinformation. The modulation recording unit ex402 electrically drives asemiconductor laser included in the optical head ex401, and modulatesthe laser light according to recorded data. The reproductiondemodulating unit ex403 amplifies a reproduction signal obtained byelectrically detecting the reflected light from the recording surfaceusing a photo detector included in the optical head ex401, anddemodulates the reproduction signal by separating a signal componentrecorded on the recording medium ex215 to reproduce the necessaryinformation. The buffer ex404 temporarily holds the information to berecorded on the recording medium ex215 and the information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 moves the opticalhead ex401 to a predetermined information track while controlling therotation drive of the disk motor ex405 so as to follow the laser spot.The system control unit ex407 controls overall the informationreproducing/recording unit ex400. The reading and writing processes canbe implemented by the system control unit ex407 using variousinformation stored in the buffer ex404 and generating and adding newinformation as necessary, and by the modulation recording unit ex402,the reproduction demodulating unit ex403, and the servo control unitex406 that record and reproduce information through the optical headex401 while being operated in a coordinated manner. The system controlunit ex407 includes, for example, a microprocessor, and executesprocessing by causing a computer to execute a program for read andwrite.

Although the optical head ex401 irradiates a laser spot in thedescription, it may perform high-density recording using near fieldlight.

FIG. 24 illustrates the recording medium ex215 that is the optical disk.On the recording surface of the recording medium ex215, guide groovesare spirally formed, and an information track ex230 records, in advance,address information indicating an absolute position on the diskaccording to change in a shape of the guide grooves. The addressinformation includes information for determining positions of recordingblocks ex231 that are a unit for recording data. Reproducing theinformation track ex230 and reading the address information in anapparatus that records and reproduces data can lead to determination ofthe positions of the recording blocks. Furthermore, the recording mediumex215 includes a data recording area ex233, an inner circumference areaex232, and an outer circumference area ex234. The data recording areaex233 is an area for use in recording the user data. The innercircumference area ex232 and the outer circumference area ex234 that areinside and outside of the data recording area ex233, respectively arefor specific use except for recording the user data. The informationreproducing/recording unit 400 reads and writes coded audio, coded videodata, or multiplexed data obtained by multiplexing the coded audio andvideo data, from and on the data recording area ex233 of the recordingmedium ex215.

Although an optical disk having a layer, such as a DVD and a BD isdescribed as an example in the description, the optical disk is notlimited to such, and may be an optical disk having a multilayerstructure and capable of being recorded on a part other than thesurface. Furthermore, the optical disk may have a structure formultidimensional recording/reproduction, such as recording ofinformation using light of colors with different wavelengths in the sameportion of the optical disk and for recording information havingdifferent layers from various angles.

Furthermore, a car ex210 having an antenna ex205 can receive data fromthe satellite ex202 and others, and reproduce video on a display devicesuch as a car navigation system ex211 set in the car ex210, in thedigital broadcasting system ex200. Here, a configuration of the carnavigation system ex211 will be a configuration, for example, includinga GPS receiving unit from the configuration illustrated in FIG. 22. Thesame will be true for the configuration of the computer ex111, thecellular phone ex114, and others.

FIG. 25A illustrates the cellular phone ex114 that uses the movingpicture coding method and the moving picture decoding method describedin embodiments. The cellular phone ex114 includes: an antenna ex350 fortransmitting and receiving radio waves through the base station ex110; acamera unit ex365 capable of capturing moving and still images; and adisplay unit ex358 such as a liquid crystal display for displaying thedata such as decoded video captured by the camera unit ex365 or receivedby the antenna ex350. The cellular phone ex114 further includes: a mainbody unit including an operation key unit ex366; an audio output unitex357 such as a speaker for output of audio; an audio input unit ex356such as a microphone for input of audio; a memory unit ex367 for storingcaptured video or still pictures, recorded audio, coded or decoded dataof the received video, the still pictures, e-mails, or others; and aslot unit ex364 that is an interface unit for a recording medium thatstores data in the same manner as the memory unit ex367.

Next, an example of a configuration of the cellular phone ex114 will bedescribed with reference to FIG. 25B. In the cellular phone ex114, amain control unit ex360 designed to control overall each unit of themain body including the display unit ex358 as well as the operation keyunit ex366 is connected mutually, via a synchronous bus ex370, to apower supply circuit unit ex361, an operation input control unit ex362,a video signal processing unit ex355, a camera interface unit ex363, aliquid crystal display (LCD) control unit ex359, amodulation/demodulation unit ex352, a multiplexing/demultiplexing unitex353, an audio signal processing unit ex354, the slot unit ex364, andthe memory unit ex367.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex361 supplies the respective units withpower from a battery pack so as to activate the cell phone ex114.

In the cellular phone ex114, the audio signal processing unit ex354converts the audio signals collected by the audio input unit ex356 invoice conversation mode into digital audio signals under the control ofthe main control unit ex360 including a CPU, ROM, and RAM. Then, themodulation/demodulation unit ex352 performs spread spectrum processingon the digital audio signals, and the transmitting and receiving unitex351 performs digital-to-analog conversion and frequency conversion onthe data, so as to transmit the resulting data via the antenna ex350.Also, in the cellular phone ex114, the transmitting and receiving unitex351 amplifies the data received by the antenna ex350 in voiceconversation mode and performs frequency conversion and theanalog-to-digital conversion on the data. Then, themodulation/demodulation unit ex352 performs inverse spread spectrumprocessing on the data, and the audio signal processing unit ex354converts it into analog audio signals, so as to output them via theaudio output unit ex357.

Furthermore, when an e-mail in data communication mode is transmitted,text data of the e-mail inputted by operating the operation key unitex366 and others of the main body is sent out to the main control unitex360 via the operation input control unit ex362. The main control unitex360 causes the modulation/demodulation unit ex352 to perform spreadspectrum processing on the text data, and the transmitting and receivingunit ex351 performs the digital-to-analog conversion and the frequencyconversion on the resulting data to transmit the data to the basestation ex110 via the antenna ex350. When an e-mail is received,processing that is approximately inverse to the processing fortransmitting an e-mail is performed on the received data, and theresulting data is provided to the display unit ex358.

When video, still images, or video and audio in data communication modeis or are transmitted, the video signal processing unit ex355 compressesand codes video signals supplied from the camera unit ex365 using themoving picture coding method shown in each of embodiments (i.e.,functions as the image coding apparatus according to the aspect of thepresent invention), and transmits the coded video data to themultiplexing/demultiplexing unit ex353. In contrast, during when thecamera unit ex365 captures video, still images, and others, the audiosignal processing unit ex354 codes audio signals collected by the audioinput unit ex356, and transmits the coded audio data to themultiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the coded videodata supplied from the video signal processing unit ex355 and the codedaudio data supplied from the audio signal processing unit ex354, using apredetermined method. Then, the modulation/demodulation unit(modulation/demodulation circuit unit) ex352 performs spread spectrumprocessing on the multiplexed data, and the transmitting and receivingunit ex351 performs digital-to-analog conversion and frequencyconversion on the data so as to transmit the resulting data via theantenna ex350.

When receiving data of a video file which is linked to a Web page andothers in data communication mode or when receiving an e-mail with videoand/or audio attached, in order to decode the multiplexed data receivedvia the antenna ex350, the multiplexing/demultiplexing unit ex353demultiplexes the multiplexed data into a video data bit stream and anaudio data bit stream, and supplies the video signal processing unitex355 with the coded video data and the audio signal processing unitex354 with the coded audio data, through the synchronous bus ex370. Thevideo signal processing unit ex355 decodes the video signal using amoving picture decoding method corresponding to the moving picturecoding method shown in each of embodiments (i.e., functions as the imagedecoding apparatus according to the aspect of the present invention),and then the display unit ex358 displays, for instance, the video andstill images included in the video file linked to the Web page via theLCD control unit ex359. Furthermore, the audio signal processing unitex354 decodes the audio signal, and the audio output unit ex357 providesthe audio.

Furthermore, similarly to the television ex300, a terminal such as thecellular phone ex114 probably have 3 types of implementationconfigurations including not only (i) a transmitting and receivingterminal including both a coding apparatus and a decoding apparatus, butalso (ii) a transmitting terminal including only a coding apparatus and(iii) a receiving terminal including only a decoding apparatus. Althoughthe digital broadcasting system ex200 receives and transmits themultiplexed data obtained by multiplexing audio data onto video data inthe description, the multiplexed data may be data obtained bymultiplexing not audio data but character data related to video ontovideo data, and may be not multiplexed data but video data itself.

As such, the moving picture coding method and the moving picturedecoding method in each of embodiments can be used in any of the devicesand systems described. Thus, the advantages described in each ofembodiments can be obtained.

Furthermore, the present invention is not limited to embodiments, andvarious modifications and revisions are possible without departing fromthe scope of the present invention.

Embodiment 3

Video data can be generated by switching, as necessary, between (i) themoving picture coding method or the moving picture coding apparatusshown in each of embodiments and (ii) a moving picture coding method ora moving picture coding apparatus in conformity with a differentstandard, such as MPEG-2, MPEG-4 AVC, and VC-1.

Here, when a plurality of video data that conforms to the differentstandards is generated and is then decoded, the decoding methods need tobe selected to conform to the different standards. However, since towhich standard each of the plurality of the video data to be decodedconform cannot be detected, there is a problem that an appropriatedecoding method cannot be selected.

In order to solve the problem, multiplexed data obtained by multiplexingaudio data and others onto video data has a structure includingidentification information indicating to which standard the video dataconforms. The specific structure of the multiplexed data including thevideo data generated in the moving picture coding method and by themoving picture coding apparatus shown in each of embodiments will behereinafter described. The multiplexed data is a digital stream in theMPEG-2 Transport Stream format.

FIG. 26 illustrates a structure of the multiplexed data As illustratedin FIG. 26, the multiplexed data can be obtained by multiplexing atleast one of a video stream, an audio stream, a presentation graphicsstream (PG), and an interactive graphics stream. The video streamrepresents primary video and secondary video of a movie, the audiostream (IG) represents a primary audio part and a secondary audio partto be mixed with the primary audio part, and the presentation graphicsstream represents subtitles of the movie. Here, the primary video isnormal video to be displayed on a screen, and the secondary video isvideo to be displayed on a smaller window in the primary video.Furthermore, the interactive graphics stream represents an interactivescreen to be generated by arranging the GUI components on a screen. Thevideo stream is coded in the moving picture coding method or by themoving picture coding apparatus shown in each of embodiments, or in amoving picture coding method or by a moving picture coding apparatus inconformity with a conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1. The audio stream is coded in accordance with a standard, such asDolby-AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, and linear PCM.

Each stream included in the multiplexed data is identified by PID. Forexample, 0x1011 is allocated to the video stream to be used for video ofa movie, 0x1100 to 0x111F are allocated to the audio streams, 0x1200 to0x121F are allocated to the presentation graphics streams, 0x1400 to0x141F are allocated to the interactive graphics streams, 0x1B00 to0x1B1F are allocated to the video streams to be used for secondary videoof the movie, and 0x1A00 to 0x1A1F are allocated to the audio streams tobe used for the secondary audio to be mixed with the primary audio.

FIG. 27 schematically illustrates how data is multiplexed. First, avideo stream ex235 composed of video frames and an audio stream ex238composed of audio frames are transformed into a stream of PES packetsex236 and a stream of PES packets ex239, and further into TS packetsex237 and TS packets ex240, respectively. Similarly, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are transformed into a stream of PES packets ex242 and astream of PES packets ex245, and further into TS packets ex243 and TSpackets ex246, respectively. These TS packets are multiplexed into astream to obtain multiplexed data ex247.

FIG. 28 illustrates how a video stream is stored in a stream of PESpackets in more detail. The first bar in FIG. 28 shows a video framestream in a video stream. The second bar shows the stream of PESpackets. As indicated by arrows denoted as yy1, yy2, yy3, and yy4 inFIG. 28, the video stream is divided into pictures as I pictures, Bpictures, and P pictures each of which is a video presentation unit, andthe pictures are stored in a payload of each of the PES packets. Each ofthe PES packets has a PES header, and the PES header stores aPresentation Time-Stamp (PTS) indicating a display time of the picture,and a Decoding Time-Stamp (DTS) indicating a decoding time of thepicture.

FIG. 29 illustrates a format of TS packets to be finally written on themultiplexed data. Each of the TS packets is a 188-byte fixed lengthpacket including a 4-byte TS header having information, such as a PIDfor identifying a stream and a 184-byte TS payload for storing data. ThePES packets are divided, and stored in the TS payloads, respectively.When a BD ROM is used, each of the TS packets is given a 4-byteTP_Extra_Header, thus resulting in 192-byte source packets. The sourcepackets are written on the multiplexed data. The TP_Extra_Header storesinformation such as an Arrival_Time_Stamp (ATS). The ATS shows atransfer start time at which each of the TS packets is to be transferredto a PID filter. The source packets are arranged in the multiplexed dataas shown at the bottom of FIG. 29. The numbers incrementing from thehead of the multiplexed data are called source packet numbers (SPNs).

Each of the TS packets included in the multiplexed data includes notonly streams of audio, video, subtitles and others, but also a ProgramAssociation Table (PAT), a Program Map Table (PMT), and a Program ClockReference (PCR). The PAT shows what a PID in a PMT used in themultiplexed data indicates, and a PID of the PAT itself is registered aszero. The PMT stores PIDs of the streams of video, audio, subtitles andothers included in the multiplexed data, and attribute information ofthe streams corresponding to the PIDs. The PMT also has variousdescriptors relating to the multiplexed data. The descriptors haveinformation such as copy control information showing whether copying ofthe multiplexed data is permitted or not. The PCR stores STC timeinformation corresponding to an ATS showing when the PCR packet istransferred to a decoder, in order to achieve synchronization between anArrival Time Clock (ATC) that is a time axis of ATSs, and an System TimeClock (STC) that is a time axis of PTSs and DTSs.

FIG. 30 illustrates the data structure of the PMT in detail. A PMTheader is disposed at the top of the PMT. The PMT header describes thelength of data included in the PMT and others. A plurality ofdescriptors relating to the multiplexed data is disposed after the PMTheader. Information such as the copy control information is described inthe descriptors. After the descriptors, a plurality of pieces of streaminformation relating to the streams included in the multiplexed data isdisposed. Each piece of stream information includes stream descriptorseach describing information, such as a stream type for identifying acompression codec of a stream, a stream PID, and stream attributeinformation (such as a frame rate or an aspect ratio). The streamdescriptors are equal in number to the number of streams in themultiplexed data.

When the multiplexed data is recorded on a recording medium and others,it is recorded together with multiplexed data information files.

Each of the multiplexed data information files is management informationof the multiplexed data as shown in FIG. 31. The multiplexed datainformation files are in one to one correspondence with the multiplexeddata, and each of the files includes multiplexed data information,stream attribute information, and an entry map.

As illustrated in FIG. 31, the multiplexed data information includes asystem rate, a reproduction start time, and a reproduction end time. Thesystem rate indicates the maximum transfer rate at which a system targetdecoder to be described later transfers the multiplexed data to a PIDfilter. The intervals of the ATSs included in the multiplexed data areset to not higher than a system rate. The reproduction start timeindicates a PTS in a video frame at the head of the multiplexed data. Aninterval of one frame is added to a PTS in a video frame at the end ofthe multiplexed data, and the PTS is set to the reproduction end time.

As shown in FIG. 32, a piece of attribute information is registered inthe stream attribute information, for each PID of each stream includedin the multiplexed data. Each piece of attribute information hasdifferent information depending on whether the corresponding stream is avideo stream, an audio stream, a presentation graphics stream, or aninteractive graphics stream. Each piece of video stream attributeinformation carries information including what kind of compression codecis used for compressing the video stream, and the resolution, aspectratio and frame rate of the pieces of picture data that is included inthe video stream. Each piece of audio stream attribute informationcarries information including what kind of compression codec is used forcompressing the audio stream, how many channels are included in theaudio stream, which language the audio stream supports, and how high thesampling frequency is. The video stream attribute information and theaudio stream attribute information are used for initialization of adecoder before the player plays back the information.

In the present embodiment, the multiplexed data to be used is of astream type included in the PMT. Furthermore, when the multiplexed datais recorded on a recording medium, the video stream attributeinformation included in the multiplexed data information is used. Morespecifically, the moving picture coding method or the moving picturecoding apparatus described in each of embodiments includes a step or aunit for allocating unique information indicating video data generatedby the moving picture coding method or the moving picture codingapparatus in each of embodiments, to the stream type included in the PMTor the video stream attribute information. With the configuration, thevideo data generated by the moving picture coding method or the movingpicture coding apparatus described in each of embodiments can bedistinguished from video data that conforms to another standard.

Furthermore, FIG. 33 illustrates steps of the moving picture decodingmethod according to the present embodiment. In Step exS100, the streamtype included in the PMT or the video stream attribute informationincluded in the multiplexed data information is obtained from themultiplexed data. Next, in Step exS101, it is determined whether or notthe stream type or the video stream attribute information indicates thatthe multiplexed data is generated by the moving picture coding method orthe moving picture coding apparatus in each of embodiments. When it isdetermined that the stream type or the video stream attributeinformation indicates that the multiplexed data is generated by themoving picture coding method or the moving picture coding apparatus ineach of embodiments, in Step exS102, decoding is performed by the movingpicture decoding method in each of embodiments. Furthermore, when thestream type or the video stream attribute information indicatesconformance to the conventional standards, such as MPEG-2, MPEG-4 AVC,and VC-1, in Step exS103, decoding is performed by a moving picturedecoding method in conformity with the conventional standards.

As such, allocating a new unique value to the stream type or the videostream attribute information enables determination whether or not themoving picture decoding method or the moving picture decoding apparatusthat is described in each of embodiments can perform decoding. Even whenmultiplexed data that conforms to a different standard is input, anappropriate decoding method or apparatus can be selected. Thus, itbecomes possible to decode information without any error. Furthermore,the moving picture coding method or apparatus, or the moving picturedecoding method or apparatus in the present embodiment can be used inthe devices and systems described above.

Embodiment 4

Each of the moving picture coding method, the moving picture codingapparatus, the moving picture decoding method, and the moving picturedecoding apparatus in each of embodiments is typically achieved in theform of an integrated circuit or a Large Scale Integrated (LSI) circuit.As an example of the LSI, FIG. 34 illustrates a configuration of the LSIex500 that is made into one chip. The LSI ex500 includes elements ex501,ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 to bedescribed below, and the elements are connected to each other through abus ex510. The power supply circuit unit ex505 is activated by supplyingeach of the elements with power when the power supply circuit unit ex505is turned on.

For example, when coding is performed, the LSI ex500 receives an AVsignal from a microphone ex117, a camera ex113, and others through an AVIO ex509 under control of a control unit ex501 including a CPU ex502, amemory controller ex503, a stream controller ex504, and a drivingfrequency control unit ex512. The received AV signal is temporarilystored in an external memory ex511, such as an SDRAM. Under control ofthe control unit ex501, the stored data is segmented into data portionsaccording to the processing amount and speed to be transmitted to asignal processing unit ex507. Then, the signal processing unit ex507codes an audio signal and/or a video signal. Here, the coding of thevideo signal is the coding described in each of embodiments.Furthermore, the signal processing unit ex507 sometimes multiplexes thecoded audio data and the coded video data, and a stream IO ex506provides the multiplexed data outside. The provided multiplexed data istransmitted to the base station ex107, or written on the recordingmedium ex215. When data sets are multiplexed, the data should betemporarily stored in the buffer ex508 so that the data sets aresynchronized with each other.

Although the memory ex511 is an element outside the LSI ex500, it may beincluded in the LSI ex500. The buffer ex508 is not limited to onebuffer, but may be composed of buffers. Furthermore, the LSI ex500 maybe made into one chip or a plurality of chips.

Furthermore, although the control unit ex501 includes the CPU ex502, thememory controller ex503, the stream controller ex504, the drivingfrequency control unit ex512, the configuration of the control unitex501 is not limited to such. For example, the signal processing unitex507 may further include a CPU. Inclusion of another CPU in the signalprocessing unit ex507 can improve the processing speed. Furthermore, asanother example, the CPU ex502 may serve as or be a part of the signalprocessing unit ex507, and, for example, may include an audio signalprocessing unit. In such a case, the control unit ex501 includes thesignal processing unit ex507 or the CPU ex502 including a part of thesignal processing unit ex507.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, ways to achieve integration are not limited to the LSI, and aspecial circuit or a general purpose processor and so forth can alsoachieve the integration. Field Programmable Gate Array (FPGA) that canbe programmed after manufacturing LSIs or a reconfigurable processorthat allows re-configuration of the connection or configuration of anLSI can be used for the same purpose.

In the future, with advancement in semiconductor technology, a brand-newtechnology may replace LSI. The functional blocks can be integratedusing such a technology. The possibility is that the present inventionis applied to biotechnology.

Embodiment 5

When video data generated in the moving picture coding method or by themoving picture coding apparatus described in each of embodiments isdecoded, compared to when video data that conforms to a conventionalstandard, such as MPEG-2, MPEG-4 AVC, and VC-1 is decoded, theprocessing amount probably increases. Thus, the LSI ex500 needs to beset to a driving frequency higher than that of the CPU ex502 to be usedwhen video data in conformity with the conventional standard is decoded.However, when the driving frequency is set higher, there is a problemthat the power consumption increases.

In order to solve the problem, the moving picture decoding apparatus,such as the television ex300 and the LSI ex500 is configured todetermine to which standard the video data conforms, and switch betweenthe driving frequencies according to the determined standard. FIG. 35illustrates a configuration ex800 in the present embodiment. A drivingfrequency switching unit ex803 sets a driving frequency to a higherdriving frequency when video data is generated by the moving picturecoding method or the moving picture coding apparatus described in eachof embodiments. Then, the driving frequency switching unit ex803instructs a decoding processing unit ex801 that executes the movingpicture decoding method described in each of embodiments to decode thevideo data. When the video data conforms to the conventional standard,the driving frequency switching unit ex803 sets a driving frequency to alower driving frequency than that of the video data generated by themoving picture coding method or the moving picture coding apparatusdescribed in each of embodiments. Then, the driving frequency switchingunit ex803 instructs the decoding processing unit ex802 that conforms tothe conventional standard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 in FIG. 34.Here, each of the decoding processing unit ex801 that executes themoving picture decoding method described in each of embodiments and thedecoding processing unit ex802 that conforms to the conventionalstandard corresponds to the signal processing unit ex507 in FIG. 34. TheCPU ex502 determines to which standard the video data conforms. Then,the driving frequency control unit ex512 determines a driving frequencybased on a signal from the CPU ex502. Furthermore, the signal processingunit ex507 decodes the video data based on the signal from the CPUex502. For example, the identification information described inEmbodiment 3 is probably used for identifying the video data. Theidentification information is not limited to the one described inEmbodiment 3 but may be any information as long as the informationindicates to which standard the video data conforms. For example, whenwhich standard video data conforms to can be determined based on anexternal signal for determining that the video data is used for atelevision or a disk, etc., the determination may be made based on suchan external signal. Furthermore, the CPU ex502 selects a drivingfrequency based on, for example, a look-up table in which the standardsof the video data are associated with the driving frequencies as shownin FIG. 37. The driving frequency can be selected by storing the look-uptable in the buffer ex508 and in an internal memory of an LSI, and withreference to the look-up table by the CPU ex502.

FIG. 36 illustrates steps for executing a method in the presentembodiment. First, in Step exS200, the signal processing unit ex507obtains identification information from the multiplexed data. Next, inStep exS201, the CPU ex502 determines whether or not the video data isgenerated by the coding method and the coding apparatus described ineach of embodiments, based on the identification information. When thevideo data is generated by the moving picture coding method and themoving picture coding apparatus described in each of embodiments, inStep exS202, the CPU ex502 transmits a signal for setting the drivingfrequency to a higher driving frequency to the driving frequency controlunit ex512. Then, the driving frequency control unit ex512 sets thedriving frequency to the higher driving frequency. On the other hand,when the identification information indicates that the video dataconforms to the conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1, in Step exS203, the CPU ex502 transmits a signal for setting thedriving frequency to a lower driving frequency to the driving frequencycontrol unit ex512. Then, the driving frequency control unit ex512 setsthe driving frequency to the lower driving frequency than that in thecase where the video data is generated by the moving picture codingmethod and the moving picture coding apparatus described in each ofembodiment.

Furthermore, along with the switching of the driving frequencies, thepower conservation effect can be improved by changing the voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500. Forexample, when the driving frequency is set lower, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set to a voltage lower than that in the case where the drivingfrequency is set higher.

Furthermore, when the processing amount for decoding is larger, thedriving frequency may be set higher, and when the processing amount fordecoding is smaller, the driving frequency may be set lower as themethod for setting the driving frequency. Thus, the setting method isnot limited to the ones described above. For example, when theprocessing amount for decoding video data in conformity with MPEG-4 AVCis larger than the processing amount for decoding video data generatedby the moving picture coding method and the moving picture codingapparatus described in each of embodiments, the driving frequency isprobably set in reverse order to the setting described above.

Furthermore, the method for setting the driving frequency is not limitedto the method for setting the driving frequency lower. For example, whenthe identification information indicates that the video data isgenerated by the moving picture coding method and the moving picturecoding apparatus described in each of embodiments, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set higher. When the identification information indicates thatthe video data conforms to the conventional standard, such as MPEG-2,MPEG-4 AVC, and VC-1, the voltage to be applied to the LSI ex500 or theapparatus including the LSI ex500 is probably set lower. As anotherexample, when the identification information indicates that the videodata is generated by the moving picture coding method and the movingpicture coding apparatus described in each of embodiments, the drivingof the CPU ex502 does not probably have to be suspended. When theidentification information indicates that the video data conforms to theconventional standard, such as MPEG-2, MPEG-4 AVC, and VC-1, the drivingof the CPU ex502 is probably suspended at a given time because the CPUex502 has extra processing capacity. Even when the identificationinformation indicates that the video data is generated by the movingpicture coding method and the moving picture coding apparatus describedin each of embodiments, in the case where the CPU ex502 has extraprocessing capacity, the driving of the CPU ex502 is probably suspendedat a given time. In such a case, the suspending time is probably setshorter than that in the case where when the identification informationindicates that the video data conforms to the conventional standard,such as MPEG-2, MPEG-4 AVC, and VC-1.

Accordingly, the power conservation effect can be improved by switchingbetween the driving frequencies in accordance with the standard to whichthe video data conforms. Furthermore, when the LSI ex500 or theapparatus including the LSI ex500 is driven using a battery, the batterylife can be extended with the power conservation effect.

Embodiment 6

There are cases where a plurality of video data that conforms todifferent standards, is provided to the devices and systems, such as atelevision and a cellular phone. In order to enable decoding theplurality of video data that conforms to the different standards, thesignal processing unit ex507 of the LSI ex500 needs to conform to thedifferent standards. However, the problems of increase in the scale ofthe circuit of the LSI ex500 and increase in the cost arise with theindividual use of the signal processing units ex507 that conform to therespective standards.

In order to solve the problem, what is conceived is a configuration inwhich the decoding processing unit for implementing the moving picturedecoding method described in each of embodiments and the decodingprocessing unit that conforms to the conventional standard, such asMPEG-2, MPEG-4 AVC, and VC-1 are partly shared. Ex900 in FIG. 38A showsan example of the configuration. For example, the moving picturedecoding method described in each of embodiments and the moving picturedecoding method that conforms to MPEG-4 AVC have, partly in common, thedetails of processing, such as entropy coding, inverse quantization,deblocking filtering, and motion compensated prediction. The details ofprocessing to be shared probably include use of a decoding processingunit ex902 that conforms to MPEG-4 AVC. In contrast, a dedicateddecoding processing unit ex901 is probably used for other processingunique to an aspect of the present invention. The decoding processingunit for implementing the moving picture decoding method described ineach of embodiments may be shared for the processing to be shared, and adedicated decoding processing unit may be used for processing unique tothat of MPEG-4 AVC.

Furthermore, ex1000 in FIG. 38B shows another example in that processingis partly shared. This example uses a configuration including adedicated decoding processing unit ex1001 that supports the processingunique to an aspect of the present invention, a dedicated decodingprocessing unit ex1002 that supports the processing unique to anotherconventional standard, and a decoding processing unit ex1003 thatsupports processing to be shared between the moving picture decodingmethod according to the aspect of the present invention and theconventional moving picture decoding method. Here, the dedicateddecoding processing units ex1001 and ex1002 are not necessarilyspecialized for the processing according to the aspect of the presentinvention and the processing of the conventional standard, respectively,and may be the ones capable of implementing general processing.Furthermore, the configuration of the present embodiment can beimplemented by the LSI ex500.

As such, reducing the scale of the circuit of an LSI and reducing thecost are possible by sharing the decoding processing unit for theprocessing to be shared between the moving picture decoding methodaccording to the aspect of the present invention and the moving picturedecoding method in conformity with the conventional standard.

INDUSTRIAL APPLICABILITY

The image decoding method and the image coding method according to thepresent invention obtain an effect of improving both image quality andcoding efficiency, and can be applied to a video camera, a mobile phonehaving functions of capturing moving pictures and reproducing movingpictures, a personal computer, or a recording and reproducing apparatus.

REFERENCE SIGNS LIST

-   1000 Image coding apparatus-   1100 Coding processing unit-   1101 Subtractor-   1102 Orthogonal transform unit-   1103 Quantization unit-   1104 Entropy coding unit-   1105 Inverse quantization unit-   1106 Inverse orthogonal transform unit-   1107 Adder-   1108 Deblocking filter-   1109 Memory-   1110 Intra prediction unit-   1111 Motion compensation unit-   1112 Motion estimation unit-   1113 Switch-   1200 Coding control unit-   2000 Image decoding apparatus-   2100 Coding processing unit-   2101 Entropy decoding unit-   2102 Inverse quantization unit-   2103 Inverse orthogonal transform unit-   2104 Adder-   2105 Deblocking filter-   2106 Memory-   2107 Intra prediction unit-   2108 Motion compensation unit-   2109 Switch-   2200 Decoding control unit

1. An image decoding method for decoding a coded stream which includes aplurality of processing units and a header for the processing units, thecoded stream being generated by coding a moving picture, the processingunits including at least one processing unit layered to be split into aplurality of smaller processing units, the image decoding methodcomprising: specifying a hierarchical layer having a processing unit inwhich a parameter necessary for decoding is stored, by parsing hierarchydepth information stored in the header; and decoding the processing unitusing the parameter stored in the processing unit located at thespecified hierarchical layer.
 2. The image decoding method according toclaim 1, wherein the coded stream is generated by coding which includesorthogonal transform and quantization, the processing unit is layered tobe smaller in a direction from a higher level to a lower level, a codingunit exists as a largest processing unit at a highest hierarchicallayer, and a transform unit exists as a processing unit smaller than thecoding unit at a lower hierarchical layer that is deeper than thehighest hierarchical layer, the parameter is a quantization parameterapplied to the transform unit, the hierarchy depth information indicatesa lower hierarchical layer that is deeper than the highest hierarchicallayer, and (i) a hierarchical layer indicated by the hierarchy depthinformation or (ii) a hierarchical layer which is higher than thehierarchical layer and is other than the highest hierarchical layer isspecified, when specifying a hierarchical layer having a processing unitin which the quantization parameter is stored.
 3. The image decodingmethod according to claim 1, wherein the header is a picture header fora picture including the processing units, and the hierarchy depthinformation is stored in the picture header.
 4. The image decodingmethod according to claim 2, wherein when the processing unit isdecoded, the quantization parameter located, within the processing unit,after a transform coefficient generated by the orthogonal transform andquantization is used.
 5. An image coding method for generating, bycoding a moving picture, a coded stream which includes a plurality ofprocessing units and a header for the processing units, the processingunits including at least one processing unit layered to be split into aplurality of smaller processing units, the image coding methodcomprising: coding the moving picture; writing, into the header,hierarchy depth information for specifying a hierarchical layer having aprocessing unit in which a parameter necessary for decoding is stored;and writing the parameter into the processing unit located at thehierarchical layer specified by the hierarchy depth information.
 6. Theimage coding method according to claim 5, wherein orthogonal transformand quantization are performed on the moving picture when the movingpicture is coded, the processing unit is layered to be smaller in adirection from a higher level to a lower level, a coding unit exists asa largest processing unit at a highest hierarchical layer, and atransform unit exists as a processing unit smaller than the coding unitat a lower hierarchical layer that is deeper than the highesthierarchical layer, the parameter is a quantization parameter applied tothe transform unit, the hierarchy depth information indicates ahierarchical layer that is lower than the highest hierarchical layer,and the parameter is written into a processing unit at (i) ahierarchical layer indicated by the hierarchy depth information or (ii)a hierarchical layer which is higher than the hierarchical layer and isother than the highest hierarchical layer, when the quantizationparameter is written.
 7. The image coding method according to claim 5,wherein the header is a picture header for a picture including theprocessing units, and the hierarchy depth information is written intothe picture header when the hierarchy depth information is written. 8.The image coding method according to claim 6, wherein, when thequantization parameter is written, the quantization parameter is writtenwithin the processing unit, after a transform coefficient generated bythe orthogonal transform and quantization.
 9. An image decodingapparatus which decodes a coded stream using the image decoding methodaccording to claim
 1. 10. An image coding apparatus which codes a movingpicture using the image coding method according to claim
 5. 11. Anon-transitory computer-readable recording medium having a programrecorded thereon for causing a computer to decode a coded stream usingthe image decoding method according to claim
 1. 12. A non-transitorycomputer-readable recording medium having a program recorded thereon forcausing a computer to code a moving picture using the image codingmethod according to claim
 5. 13. An integrated circuit which decodes acoded stream using the image decoding method according to claim
 1. 14.An integrated circuit which codes a moving picture using the imagecoding method according to claim 5.